Skip to main content

Incident Report: Persisting Genuine Stuck Collector Alert

Date: 2023-09-30
Time: 23:49 (GMT+5:30)
Duration: 01:15 hours:minutes

Description

Persisting Genuine Stuck Collector Alert on fantom (on staging as well as production): Detected stuck collector on fantom: current block (68778020) and last queried block (68777528) recorded on 2023-09-29T18:02:46.085.

Root Cause

Problem with the deployment happened earlier that day.

Impact

Low impact on devs.

Timeline

  • 23:50 - First noticed the issue.
  • 12:00 - Initial diagnosis.
  • 12:40 - Started the fix.
  • 01:02 - Issue resolved.

Lessons Learned

Deployments needs to be checked and properly monitored during the fixes.

Actions Taken

Aaron redeployed event collectors.

Incident Reviewer(s)

  • Andrew
  • Aaron